This application is based on Japanese Patent Application No. 226677/1998 filed on Aug. 11, 1998 and No. 20621/1999 filed on Jan. 28, 1999, the contents of which are incorporated herein by reference.
1. Field of the Invention
The present invention relates to a method and a device for providing information used in the field of virtual reality (VR) or computer visualization. More specifically, the present invention relates to a method and a device for interactively providing a virtual object without inconsistency with a real object concerning their shielding relationship.
2. Description of the Prior Art
In recent years, researches about engineering applications of the artificial reality by positively fusing the physical space and the information space have been activated, which include a study about the complex reality. However, most of the researches utilize a head mounted display (HMD) or a normal projector. In this case, their reality can be substantially deteriorated due to an inconsistency between the real object and the virtual object concerning their shielding relationship.
In addition, when providing not only for the visual sense but also the tactile sense including interaction with a virtual object, as disclosed in Japanese Unexamined Patent Publication No. 8-257947(A), for example, the tactile sense provision device can interfere with the visual sense provision device visually and physically. This is a large problem for realizing a system that provides the visual sense and the tactile sense simultaneously.
Reality Fusing
If the physical space where a human is living and the information space that is structured by computers and communication networks are fused seamlessly, the society will be more highly computerized or information-based. To realize the more highly information-based society, a concept such as a wearable computing or a ubiquitous computing is proposed.
Here, the meaning of seamless fusion will be investigated. In the field of psychology, etc., a process is called xe2x80x9cfusionxe2x80x9d that gives a novel sense image having a close relationship with plural information and not being generated by itself toward a certain phenomenon or an object, as shown in binocular fusion, for example.
In the engineering field too, a sensor fusion by integrally processing information obtained by plural sensors is being researched, which will provide a novel sensing function that is not obtained from a single sensor.
In the same way, it is expected to obtain a new effect that is not obtained from a single space, as a result of not only a simple mixture of such spaces as the physical space where a human exists and the information space of computers or the visual space and the tactile space but also the seamless fusion thereof.
Fusion of the Physical Space and the Information Space
Along with building up of the information and communication infrastructure, it is considered that on demand type and bidirectional type communications will be the mainstream instead of the conventional broadcast type communication. Moreover, for an application that requires higher interactivity, it is considered that the information will be more personalized and the personal reality fusion of the personal information will be necessary.
Therefore, as a personal interface, this reality fusion should be realized at low cost and compactly.
However, the conventional display, in which the device itself occupies a large percentage of weight and volume, is not suitable for a personal application.
Compared with the conventional display mentioned above, a head mounted display (HMD) that is used in a field of virtual reality recently is suitable for creating a personal information space. However, the HMD is hard to be used in the physical space since the display can screen off the physical space, so that a work of an operator in the physical space is very difficult. Therefore, the image of his or her own body can be seen only as a metaphor by a computer graphics (CG), resulting in low ability of self-projection that is considered an important factor of the reality.
Another type of HMD, i.e., a see-through type HMD is also proposed, in which an image of the information space is superimposed on an image of the physical space using a half mirror or a head mounted camera. This has a principle disadvantage of working such as an inconsistency of the shielding relationship between provided objects or that of the focal points in the work space portion since the composition of images is performed in a position adjacent to the observer""s eyes or in the computer. The problem concerning the inconsistency of the shielding relationship can occur also in an immersive projection display such as a CAVE (Carolina Cruz-Neira et al. xe2x80x9cSurround-Screen Projection-Based Virtual Reality: The Design and Implementation of the CAVExe2x80x9d COMPUTER GRAPHICS Proceedings, Annual Conference Series, pp. 135-142, 1993) or a CABIN (Hirose et al. xe2x80x9cA Development of Immersive Multiscreen Display (CABIN)xe2x80x9d, Japan Virtual Reality Institute Second Convention Paper, pp. 137-140, 1997, which have been widely used these days.
Display Fusing Visual and Tactile Senses
It is reported that presence and workability are improved by providing an object in the information space including not only visual information but also tactile information.
However, when trying to provide the visual information and the tactile information simultaneously, the visual information provision device and the tactile information provision device interfered with each other optically and geometrically. This problem is similar to that in fusing the physical space and the information space, which is caused by the fact that the providing portion of the interface exists in the physical space. Concerning this problem, some studies have been reported up to now, for example, in WYSIWYF display (Yokokohji et al. xe2x80x9cWhat You Can See is What You Can Feel-Development of a Visual/Haptic Interface to Virtual Environmentxe2x80x9d, Proceedings of VRAIS ""96, pp. 46-53, 1996, PDDM (Noma et al. xe2x80x9cHaptic Interface for Direct Manipulation with Palmtop Displayxe2x80x9d, The eleventh Human Interface Symposium Papers pp. 491-496, 1995), and Haptic Screen (Iwata et al. xe2x80x9cHaptic Screen,xe2x80x9d Japan Virtual Reality Institute First Convention Papers, pp. 7-10, 1996). However, in most cases, the problem of the shielding relationship is still ignored or it is addressed by optical superimposition using a half mirror.
Reality Fusion Structure
Here, it is supposed that most of the problems in the conventional system are caused by the fact that the physical space and the information space are composed outside the point where the both objects of the physical space and the information space are to be interacted. In other words, it is extremely difficult for the conventional technology to cover all of the reality fusion from short distance to long distance by a single display. To solve this problem, it may be better to use different distances and sizes of the devices in accordance with the application.
From the viewpoint of the above-mentioned recognition, the inventor has proposed an object-oriented display, which is suitable for observing and operating a virtual object on hands (Japanese Patent Application No. 10-172722). The object of the present invention is to enhance the concept and to realize seamless reality fusion in wider area by xe2x80x9cdisplaying close objects by a near display while displaying far objects (landscape) by a distant displayxe2x80x9d similarly to flats on a stage.
This method is also considered as an implement in the real world of the three-dimensional virtual space generation method by flats method in the image based rendering field (Ozaki et al., xe2x80x9cTele-existence from Virtual Viewpointsxe2x80x94Creating Virtual Space Using Scenery Planesxe2x80x9d, Thirty-fifth Instrumentation and Automatic Controls Institute Scientific Lecture Papers, pp. 209-210, 1996, Hirose et al., xe2x80x9c3-D Virtual Space Generation by using 2-D Photo Imagesxe2x80x9d, Generation of the Three-Dimensional Virtual Space by Using the Two-Dimensional Real Image,xe2x80x9d Instrumentation and Automatic Controls Institute, Human Interface N and R, Vol. 11, No. 2, pp. 209-214, 1996).
However, in order to perform the reality fusion properly in a wide area, it is required to dispose displays unevenly in the wide area in accordance with the display area. In other words, it is necessary to construct a so-called ubiquitous display. The conventional displays are impractical because they require huge space and cost.
The conventional displays are effective for broadcasting type information provision, which continuously provides one-way and non-directional information toward a space including scores of observers. However, when the conventional display is used as a personal information provision device in the interactive application, i.e., used for a specific observer, the information is transmitted to the direction where no observer exists since it is a non-directional display, resulting in a waste. In addition, if the information is observed by people other than the aimed observer, it is difficult to protect privacy of individuals.
Namely, as an interactive image display device, a device xe2x80x9cobserveexe2x80x9d is needed that is observed by a human rather than a display device for displaying an image for the human.
This xe2x80x9cobserveexe2x80x9d may be constructed as a system for providing an image in the area where the observer can see. The system does not need to provide an image in the area that is shielded by an object or the back surface that cannot be seen by the observer.
In order to realize the function of the observed information provision device, xe2x80x9cobservee,xe2x80x9d an image provision system that can form a projection volume which agrees with the viewing volume that is the view area of a human may be promising.
There is a system performing the above-mentioned function, which provides an image only to the observer by using an optical system such as lenses so that the image follows the observer, whose position is previously measured. There is another system, which projects an image from the view point that is conjugated to the observer to a retroreflection screen that reflects a light ray to the incident direction.
However, the former observer following type system is difficult to use for a simultaneous and unevenly existing application since the display device requires much cost. The latter projection type image provision system has an advantage in that a lot of display screens with light weight can be produced very easily by integrating space and cost for displaying image in the projector side. It has another advantage in that the limited band of the communication infrastructure can be used effectively by providing information necessary for the observer at the necessary level. In addition, since people other than the observer cannot see the image, privacy of the observer can be protected.
Furthermore, in the situation where one or more displays are required for one person, it is better in cost performance to add the information generation function not to the space side but to the person side.
Furthermore, the observer following system can cause problems such as focus depth due to variation of the screen position and light quantity. Since a display device for displaying an image for a human is usually aimed at providing information in a wide area, one target of the research is to widen the view angle of the display.
The conventional screen is not the exception. The object of the device is to diffuse the incident light by using glass beads. Namely, a screen used for the xe2x80x9cdisplayxe2x80x9d in a movie theater, for example, has an essential characteristic that the light quantity varies inversely with the square of the distance.
Therefore, the necessary function of the screen to realize the xe2x80x9cobserveexe2x80x9d is not to diffuse the incident light but to reflect the incident light retrorsely in the direction of the observer.
Furthermore, the xe2x80x9cobserveexe2x80x9d can provide unique image information to each person. One physical device can provide unique image information to each person seeing the device. Thus, the space can be used more efficiently.
Though the inventor has proposed a method for displaying image to realize the xe2x80x9cobserveexe2x80x9d in Japanese Patent Application No. 10-172722, it has a problem as follows.
When providing a sense information except the visual sense information, especially a tactile sense information to the observer in the virtual space, the device for providing the tactile information should be disposed in front of the observer, for example. In some cases, a human state measuring sensor such as a three-dimensional position sensor may exist between the eyes of the observer and the displayed image.
If the object that is not to be seen in the virtual space is seen by the observer, the reality of the virtual environment can be deteriorated. Especially, if the object that is not to be seen in front of the displayed image is seen, the object produces shielding of the visual sense, and the displayed image has a blind spot. Thus, the reality of the image is substantially deteriorated.
A critical case of this problem can occur when the front and rear relationship between the virtual object and the real object becomes opposite. In this case, transparent view interpretation is generated, in which the object is seen as transparent, resulting in substantial deterioration of the reality.
For example, FIG. 5(A) shows a natural relationship that a virtual ball BL1 is seen in front of the hands HD of a real human, while FIG. 5(B) shows the inverse relationship concerning front and rear of the ball BL1 and the hands HD, generating inconsistency of the shielding relationship, which is not natural.
Therefore, it is important to display an image so that a real object to be not seen existing between the observer and the displayed image cannot be seen, while another object to be seen can be seen, thus any inconsistency cannot occur concerning the shielding relationship between the real object and the virtual object.
The object of the present invention is to provide a method and a device for providing information, which can provide the correct shielding relationship between the real object and the virtual object, as well as a virtual space with a high reality.
A method of the first aspect of the present invention, as shown in FIGS. 1 and 2, provides information of the virtual space to an observer BS by projecting an image, toward screens 11 and 11A having retroreflection function, from projectors 14 and 14A disposed at the position that is optically conjugated to the position of eyes of the observer BS. If real objects JT, e.g., tactile sense provision device 16 and 16Z, which are to be hidden by the virtual space, exist, a part or the whole of the real objects JT, i.e. 16 and 16Z, are optically hidden from a projection by the projectors 14 and 14A by a shielding member having retroreflection function, e.g., by disposing a shielding screen 18 on which a retroreflection material is coated, or by coating a retroreflection material 18A and 18Z.
A method of the second aspect of the present invention hides the real object JT by coating a retroreflection material on the real object JT.
A method of the third aspect of the present invention hides the real object JT by disposing a shielding screen 18 having retroreflection function in front of the real object JT.
In a method of the fourth aspect of the present invention, a tactile sense provision device 16 is provided for providing a tactile sense to the observer BS. The tactile sense provision device 16 is hidden by disposing a shielding member such as a shielding screen 18 on which a retroreflection material is coated or by coating a retroreflection material 18A and 18Z.
In a method of the fifth aspect of the present invention, a real object JT is disposed between the observer BS and the screens 11 and 11A, retroreflection function is added to the portion that is to shield the real object JT from the virtual space. Concerning the portion of the real object JT to be shielded, an image that is projected to the portion of the real object JT to be shielded is provided to the observer instead of the image to be projected to the screen 11 and 11A.
A method of the sixth aspect of the present invention, as shown in FIG. 7, provides information of the virtual space by optically hiding a part or the whole of the real objects JT disposed between the background object 11B and the observer BS. To the portion of the real objects JT to be hidden, the image of the background object 11B corresponding to the portion is projected by the projector 14B disposed at the position optically conjugated to the position of eyes of the observer BS. It is preferable that the portion of the real objects JT to be hidden is covered with a retroreflection member 18B or a retroreflection material 18Ba is coated as shown in FIG. 9.
A device of the seventh aspect of the present invention includes a projector 14A for projecting an image disposed at the position that is optically conjugated to the position of eyes of the observer, an image generating device 13A for generating an image of an object to be observed by the observer BS and outputting the image to the projector 14A, a screen 11A having retroreflection function for displaying the image projected by the projector 14A, and a tactile sense provision device 16 for providing tactile sense to the observer BS. The tactile sense provision device 16 is shielded optically from the projection by the projector 14A by a shielding member having retroreflection function, e.g., by a retroreflection screen 18 on which a retroreflection material is coated or by coating a retroreflection material 18A and 18Z.